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Community College Pathways: Summative Assessments 


EXECUTIVE SUMMARY 

Carnegie's Community College Pathways (CCP) offers two pathways, Statway® and Quantway®, that 
reduce the amount of time required to complete developmental mathematics and earn college-level 
mathematics credit. The Pathways aim to improve student success in mathematics while maintaining 
rigorous content, pedagogy, and learning outcomes. 

It is essential that the Pathways' success in securing college credit for participants does not come at the 
expense of the rigor of the curriculum or level of student learning. We have previously documented that 
Pathways students successfully complete their developmental mathematics requirements and earn 
college credit at rates two or three times higher than is typical, and do so in roughly half the time 
(Strother et al, 2013; Van Campen et al, 2013). To fully investigate the effectiveness of the Pathways, we 
must now address the question of whether successful completion of a Pathway also represents rigorous 
high levels of learning. 

The use of a common independent measure of student learning, the Pathways summative assessments 
allowed such investigation in two ways. First, the manner in which the summative assessment systems 
were developed allowed us to investigate whether Pathways students acquired a level of knowledge 
comparable to that of students who took other mathematics or statistics courses during college. Second, 
correlational analyses between course grades and the summative assessments allowed exploration of 
whether those grades were positively associated with higher performance on the independent 
measure. 

This report describes the comparative analyses from the Pathways summative assessment system. All 
items on the summative assessments were rigorously validated by field testing with over 500 members 
of a comparison sample who had taken a college mathematics or statistics course. Subsequently, all 
Pathways students took the summative assessments at the end of each term. This report describes 
analyses that use the development sample for comparison to Pathways students to assess their relative 
levels of knowledge. 

This report then describes the relationship between students' summative assessments and their end-of- 
term grades. This analysis is to provide further evidence that Pathways students are not just receiving 
high grades and successfully completing the Pathways at a higher rate, but that their grades are 
reflective of actual learning, as corroborated by the independent measure. 

Key Findings 

• Pathways students acquire a level of knowledge comparable to students who take other mathematics 
or statistics courses during college. We analyzed how Pathways students performed on each item of 
the summative assessment compared to the comparison sample. In the aggregate and on the vast 
majority of the array of concepts that were tested across two years of Pathways summative 
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assessments, the Pathways students consistently performed at or above the level of the comparison 
sample. 

• Summative assessment scores are related to students' end-of-term grades. During each term of the 
first two years of the Pathways, students' grades were significantly correlated to their performance on 
the corresponding summative assessments. These results indicate that successful students who 
received higher grades genuinely achieved higher levels of knowledge. 
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PATHWAYS SUMMATIVE ASSESSMENTS 


Purpose 

Carnegie's Community College Pathways (CCP) aim to improve the academic success rate of community 
college students in developmental mathematics. To accomplish this, there are two pathways, Statway 
and Quantway, that reduce the amount of time required to earn college credit while improving the 
content and pedagogy of developmental mathematics (Strother, Van Campen, & Grunow, 2013). While 
the Pathways' streamlined structure is a major factor in its success, it is essential that the streamlining 
does not come at the expense of the rigor of the curriculum. 

The Carnegie Foundation built a summative assessment system into the Community College Pathways to 
have a common measure that could determine the level of student knowledge across the varied 
institutions and populations of students who participated in the Pathways. The assessments measure 
Pathways students' knowledge by using a secure, independent, and common assessment of outcomes 
that have been validated as appropriate for college-level mathematics. They also allow Carnegie to 
identify and guide needed improvements in the curricula or instructional materials. The assessments, 
which are common across all classrooms, also help identify potential, positive deviants, or bright spots, 
as sources of innovation and improvement ideas. 

Carnegie wants to ensure that Pathways students are meeting the learning goals and outcomes that the 
Pathways require of them. The summative assessments allow Carnegie to ensure that Pathways 
students are completing each course with a high level of knowledge. The assessment development 
process allows Carnegie to explore if Quantway students understand quantitative reasoning and Statway 
students understand statistics at the same levels of performance as college students who have taken a 
traditional mathematics or statistics course, respectively. This report describes analyses that examine 
the performance of CCP students in comparison to students who took other college-level mathematics 
or statistics. It demonstrates that students who successfully completed a Pathways course acquired a 
level of knowledge comparable to that of college students taking other mathematics or statistics 
courses. 

Background 

Community College Pathways Networked Improvement Communities 

The academic success rate of community college students in developmental math is alarmingly low. 
There now an estimated 12.8 million students enrolled in community college (American Association of 
Community Colleges, 2014) and 59 percent of the entering students are required to take at least one 
developmental mathematics class before they are eligible to enroll in college-credit courses (Bailey, 
Jeong, and Cho 2010). However, as many as 80 percent of the students who place into developmental 
mathematics do not successfully complete any college-level mathematics courses within three years 
(Bailey et al., 2010). Instead, many students withdraw from courses, spend long periods of time 
repeating courses, or more often do not enroll or drop between courses during their lengthy 
developmental mathematics sequence, ultimately leaving college without a credential (Bailey et al., 
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2010). This means that millions of students each year fail to acquire essential mathematics skills and are 
unable to progress toward their career and life goals. 

To address this national problem, the Carnegie Foundation for the Advancement of Teaching with 
support from the Bill and Melinda Gates Foundation, William and Flora Flewlett Foundation, Kresge 
Foundation, Carnegie Corporation of New York, and the Lumina Foundation created an innovative and 
transformative approach to developmental mathematics education, the Community College Pathways. 

The CCP Networked Improvement Community is organized around two mathematics pathways, Statway 
and Quantway, that aim to facilitate students' progress through their developmental mathematics 
sequence and a college-level course for credit. The Pathways reduce the time required to earn college 
credit while improving the content and pedagogy of developmental mathematics. 

Statway is designed as a two-semester college-level statistics course with appropriate developmental 
mathematics embedded within it. This single pathway merges elementary and intermediate algebra 
with college-level statistics. Statway integrates developmental mathematics skills and college-level 
statistics into a collaborative, problem-focused class. 

Quantway is designed as two separate semester-long courses. The first semester, Quantway 1, fulfills 
the requirements for students' entire developmental mathematics sequence. The next semester, 
Quantway 2, is a college credit-bearing quantitative reasoning course. 1 Each semester of Quantway is 
designed to promote success in community college mathematics and cultivate quantitatively literate 
students. 

Learning Outcomes 

The summative assessments target key learning outcomes for each of these Pathways. In 2009 for 
Statway and 2010 for Quantway, the Carnegie Foundation convened a series of meetings to devise 
learning outcomes for each Pathway. The committee members represented the national mathematics 
and statistical societies, as well as university and community college faculty. These convenings drew 
upon participants' deep content knowledge and diverse experiences to debate and reach consensus on 
a list of core concepts and skills that mathematics students need to learn. Throughout this process, the 
committees made a deliberate effort to ensure that the Pathways' learning outcomes were comparable 
in content and rigor to those of typical college statistics (Statway) or general quantitative reasoning 
(Quantway) courses. Bernadine Chuck Fong, the senior managing partner of the Community College 
Pathways, then formally presented these learning outcomes to a number of national professional 
mathematics and statistics societies (the Mathematical Association of America, the American 
Mathematical Association of Two-Year Colleges, and the American Statistical Association) for review and 
critique. These societies subsequently validated that the learning outcomes were of equivalent quality 
and rigor as those of typical college mathematics courses. 

1 Colleges can use Quantway 1 alone to help students progress more efficiently through their developmental mathematics 
requirements, or they may offer Quantway 2 along with Quantway 1 as a full pathway option for students to earn college-level 
mathematics credit. 
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Creating Summative Assessments 

Item Generation 

The Pathways summative assessment items were created to assess student knowledge of the key 
learning outcomes. In this way, Carnegie ensured strong alignment of the Pathways' learning outcomes, 
curriculum, instruction and the assessment of those outcomes. 

As the Pathways launched, Carnegie recruited Pathways faculty members who had the best grasp of the 
curriculum and learning outcomes to draft the first set of summative assessment items. During the 
second year, a measurement specialist worked with these faculty members to create standards and 
guidelines for item writing that all item writers have since used. (Sample items are included in Appendix 
A.) To maintain a replenishing pool of items, the faculty members periodically generated new items. 
Typically once or twice a year faculty item writers wrote items that targeted specific learning outcomes 
where we needed more quality items. 

Item Reviews 

After faculty members drafted items, each item underwent several checks to ensure they were of high 
quality. First, faculty item writers reviewed one another's work. Content experts then checked each item 
for clarity, rigor, content accuracy, and appropriate matching to the intended learning outcomes. 
Additional Pathway faculty members also reviewed the items with these criteria in mind. Next, a team at 
UCLA with expertise in language and literacy development reviewed the items to ensure that they were 
appropriately written for the range of language and reading abilities within our Pathways population. 
Last, there was a final review by a measurement expert. 

Administration to Comparison Sample 

Before using any items on the summative assessments, Carnegie ensured their technical quality by field 
testing all items with a sample of individuals comparable to those who would encounter the items in the 
summative assessments. Note that the primary purpose for gathering these item testing data was to 
support the development and refinement of the assessments through psychometric analyses. Thus 
Carnegie contracted with Luth Research who recruited and administered the items to a comparable 
sample. 

To form a comparison sample, Luth Research screened its pool of users for those who were qualified to 
complete the items and offered the users a small monetary reward as compensation. Luth Research 
filtered potential survey completers to select those who best approximated the characteristics of the 
students with whom operational use was intended. For our purposes, members of the comparison 
sample had to meet the following criteria: 
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1) They reported their age in the range of 18-30 

2) They reported their highest level of education as 'some college' or 'completed college' 

3) For the question "During your time in college, which, if any, of the following courses did you 
take for at least 1 semester?", they indicated they had taken statistics (for Statway 
comparison) or mathematics (for Quantway comparison). 

Drawing on the pool of draft items, we prepared forms for administration to the comparison sample. 
Each form contained approximately 60-70 items. We chose 60-70 items per form to maximize efficiency 
and minimize 'burnout.' Luth Research then administered each form to at least 500 individuals who 
qualified for the comparison sample. 

During the first two years of the Pathways, Luth administered five forms of Statway items and two forms 
of Quantway 1 items to a total of 3,533 members of the comparison sample. For Quantway, this report 
focuses only on Quantway 1 since complete comparative data from Quantway 2 are not available. 

Item Validation 

Luth Research cleaned and sent all item test data to Carnegie. We then performed psychometric 
analyses on all items. We performed Rasch analysis, based on item response theory, to: 

• examine the reliability among items; 

• identify items that are misfit (items whose behavior does not conform the overall structure of 
the exam); 

• identify items that have problematic response patterns; and 

• establish the difficulty levels for each item. 

This analysis also allowed misfitting students (such as those who randomly guessed at items or had a 
disingenuous response pattern, such as selecting 'C' for every problem) to be identified and filtered out 
of the analyses. Following these analyses, we added those items that functioned well psychometrically 
to an item bank from which we generated the operational forms of the summative assessments 
according to carefully constructed content specifications. 

When Luth administered two forms in parallel, we embedded a number of 'anchor items' on both forms. 
We used anchor items to perform psychometric analyses on all items and persons in terms of a common 
metric. This helped control for any potential differences between groups of the comparison sample who 
completed separate forms. This process also generated more reliable item estimates and allowed us to 
create an equated item bank. We anchored between 11 percent and 42 percent of generated items 
during our various rounds of administration to members of the comparison sample. We anchored the 
smaller percentage across the first two forms in fall 2011, but then elected to anchor higher percentages 
of items as the assessment system grew and improved. The percentage was also higher in later rounds 
since we anchored items from previous forms to put new items on the same scale as previously 
validated items. We further describe this process in an upcoming technical report (see Strother & 
Yamada, in preparation). 
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Assessment Assembly and Review 

The operational forms of the common assessments were approximately 40-item measures of student 
knowledge and skill. The Pathways leaders, along with a core group of expert advisors, selected which 
learning outcomes to target on the assessments. The committee felt the selected learning outcomes 
represented a set of core competencies at an appropriate level of rigor that spanned the breadth of the 
content in the course. Carnegie's measurement specialist selected from the pool of validated items to 
match these specifications. The specialist selected items with varying levels of difficulty and good 
psychometric properties. Once we assembled an assessment, we sent it to select faculty members who 
reviewed the assessment to detect any last minute issues. Since the curriculum was improving 
continuously and since items with flaws could make it through even this extensive vetting process, a 
small percentage of items were eliminated or adjusted during this last review. 

Summative Assessment Administration 

All Statway students take a 'Mid-Pathway Assessment' at the end of the first term and an 'End-of- 
Pathway Assessment' at the end of the second term. Similarly, students in Quantway 1 take a 
'Summative Assessment' at the end of the course. We create a new assessment for each of these 
benchmark points every year. For example, students who began Statway in fall 2012 had a different 
Mid-Pathway Assessment than those that began the Pathway in fall 2011. However, up to 40 percent of 
items on each assessment are previously used so we can statistically equate all assessments. Institutions 
that implement either Pathway sign a Memorandum of Agreement with Carnegie that commits 
participation in the administration of the summative assessment. This allows Carnegie to gather data 
from every classroom across all participating faculty and institutions. 

We provided the assessments to Pathways faculty along with an answer sheet. We also provided 
guidelines for administration to faculty members to ensure uniform testing conditions to make certain 
the consistency and validity of the data. For example, the exam should be proctored, completed 
individually, and counted as part of the course grade to maintain reasonable security and ensure a good 
faith effort. Additionally, faculty members received a form to describe any deviations from the 
assessment or administration procedures prescribed. Carnegie's assessment specialist then decided if 
any aberrations warranted that the data be excluded from future Pathway-wide analyses. Two 
researchers from Carnegie independently entered all students' responses into a spreadsheet. We 
compared these files and rectified all discrepancies, compiling all data into one database. 
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COMPARING PATHWAYS STUDENTS TO THE COMPARISON SAMPLE 
Comparative Analyses: Goal and Limitations 

The analyses below explored whether the Pathways students attained similar levels of knowledge and 
skill as college students who have taken other mathematics or statistics courses. The analyses compared 
the knowledge of Pathways students at the end of each course to that of the comparison sample who 
had taken at least one mathematics or statistics course while in college. 

The analyses had several limitations that one should consider when interpreting the results. The explicit 
purpose of collecting the comparison data was to develop and refine the assessments. The ability to 
compare the performance of Pathways students and the comparison sample on each item was an 
ancillary benefit of this process. The comparison sample was self-selecting and received compensation 
for their efforts. Monetary rewards for completion may have affected how carefully and thoughtfully 
the comparison sample completed each item. There was also lag time, potentially as long as eight or 
nine years, between completing the comparable mathematics or statistics course and taking the 
assessment items for the comparison sample. The lag time was limited by using a restricted age range, 
but we could not determine precise lag. 

Data Sources 

We analyzed Pathways students' performance against the comparison sample's performance across the 
Pathways' first two academic years. This report includes data collected from fall 2011 through spring 
2013. The data came from two sources: 

Comparison Sample 

As described above, Carnegie contracted with Luth Research who administered a total of seven forms of 
summative assessment items to 3,533 field test participants. This comparison sample excluded field test 
participants who either submitted incomplete forms or for whom the click records from the online 
platform administration suggested that they were completing the items too quickly to have even read 
them fully. Luth dropped approximately a third of the field test participants for these reasons. 

As required to qualify for survey completion, all members of the comparison sample were between 18 
and 30 years of age, reported their highest level of education as 'some college' or 'completed college', 
and indicated they had taken statistics (for Statway comparison) or mathematics (for Quantway 
comparison) during their time in college. 

Table 1 shows summary statistics for their reported highest level of education. This information is 
limited since we do not know the exact types of colleges or programs in which members of the 
comparison sample enrolled. 
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Table 1 

The highest level of education reported by the 3,533 members of the comparison sample. 



Statway 

Quantway 


Comparison Sample 

Comparison Sample 

Some College 

40.6% 

46.8% 

Completed College 

59.4% 

53.2% 


Table 2 displays summary statistics for age and other demographic data for the comparison sample and 
Pathways students. The Pathways students had a higher percentage from minority groups. The 
Pathways students more frequently grew up in homes where the language used was not English-only 
and had lower maternal education (e.g. 30 percent and 34 percent of Statway and Quantway students 
respectively reported that their mothers had a college degree compared to 46 percent of the 
comparison sample). The two groups did have comparable makeups in terms of gender. 
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Table 2 

Demographic data for Statway and Quantway students and members of the comparison sample. 



Statway 

Quantway 

Comparison 

Sample 

N a 

2,630 

1,820 

1,031 

Sex 

Female 

59.8% 

62.8% 

68.1% 

Male 

40.2% 

37.2% 

30.1% 

Age 6 

Mean 

27.6 

24.6 

24.6 

S.D. 

10.9 

9.7 

3.5 

Race 

Hispanic 

32.7% 

11.9% 

11.3% 

Caucasian 

31.7% 

46.7% 

62.6% 

African American 

24.3% 

34.7% 

7.7% 

Asian 

5.3% 

2.5% 

10.6% 

American Indian/Alaskan Native 

0.5% 

0.1% 

0.5% 

Pacific Islander 

1.2% 

0.0% 

0.2% 

Other 

0.8% 

2.4% 

0.6% 

Multiple 

3.5% 

1.7% 

6.6% 

Home Language Growing Up 

English only 

58.6% 

72.4% 

77.2% 

English and another language 

29.9% 

18.2% 

16.8% 

A non-English language only 

11.5% 

9.5% 

6.0% 

Maternal Education 

Less than high school 

14.8% 

11.8% 

8.4% 

High school graduate or GED 

28.6% 

29.9% 

12.9% 

Some college but no degree 

26.3% 

24.2% 

32.6% 

2-year college degree 

10.6% 

12.7% 

11.6% 

4-year college degree 

13.6% 

13.8% 

24.5% 

Graduate or professional degree 

6.1% 

7.6% 

10.0% 


a Pathways data reflect all students in our analytic samples from the first two years (Strother 
et al., 2013; Van Campen et al., 2013). For this comparison, we collected detailed demographic 
data from 1,031 Statway comparison sample members who completed one of two surveys in 
fall 2013. This data was not collected from the comparison sample prior to fall 2013. 

6 Age was collected from all comparison sample members who completed any of the seven 
forms described in this report (n=2530 for Statway; n=1003 for Quantway). Data were 
aggregated since comparison sample members completing each form were similar. 
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Pathways Students: Summative Assessments 

Over 3,000 different students took a summative assessment for at least one term during the first two 
years of the Pathways. Statway launched in fall 2011, so four semesters of Mid-Pathway Assessment 
data were available for analysis: fall 2011, spring 2012, fall 2012, and spring 2013. Data from the End-of- 
Pathway Assessment were available for the spring terms of both years. Quantway 1 launched in spring 
2012, so three semesters of data were available: spring 2012, fall 2012, and spring 2013. 

Pathways Students: Grades 

Following every term, institutional researchers at participating colleges provided Carnegie with the final 
course grade for each student in the Pathways. Since the assessments were required to count as part of 
the course grade, we also collected the percentage of students' grade that was accounted for by the 
summative assessment starting in fall 2012. 

Analytic Methodology 

We provide below descriptive analyses of the performance of the Pathways students and the 
comparison sample on the summative assessments. We also examined the relationship between 
students' grade and their performance on the summative assessments. 

Descriptive Analyses 

One purpose of these analyses was to quantify and compare the performance of Pathways students to 
the performance of a comparison sample on the summative assessment items. For each item on each 
Pathways summative assessment, we determined the percent of Pathways students and the comparison 
sample who answered that item correctly. We aggregated these data across the items that comprised 
each summative assessment 2 . For example, for the fall 2011 Mid-Pathway Assessment, we calculated 
the average percent of correct responses across all 44 of the assessment items. 

We also wanted to compare how Pathways students and the comparison sample performed on specific 
topical content within the broader domains. To accomplish this, we aggregated the item-level results at 
the module level. The Statway curriculum includes 12 modules and Quantway 1 includes 4 modules that 
group learning outcomes into broader topical content areas. Appendix B includes information about 
each module and Appendix C includes the descriptive results at the module level. 

We also wanted to determine if the summative assessments were consistent with faculty members' 
overall assessment of student performance in the course. We used grades as the faculty members' 
overall assessment, which typically included graded homework, quiz results, faculty members' 
judgments, and other faculty-developed tests. We performed correlational analyses to examine the 
relationship between Pathways students' grade and their summative assessment score. Since the 

2 A few items were significantly altered after they were administered to the comparison sample due to critical 
feedback during the last round of faculty review. Data from these items could not be fairly compared, so these 
items were excluded from the analyses. 
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summative assessments constituted a variable portion of the student's final grade, we also computed 
partial correlations to control for the percent of students' grade that was based on the summative 
assessment. In addition, as a further test, we ran a hierarchical linear model (HLM) analysis with 
students' assessment scores predicting their grade at level 1 and the percent of students' grade 
accounted for by the summative assessment, as reported by each faculty member, as a level-2 
predictor. 
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RESULTS 


Descriptive Analyses 

Pathways versus Comparison Sample 

Pathways students consistently performed equal to or higher on the summative assessments in relation 
to the comparison sample (see Table 3). For example, Table 3 shows that Statway students performed, 
on average, 4.9 percent higher across the items used on the fall 2011 Mid-Pathway Assessment. In fact, 
the Pathways students demonstrated a higher percent of correct responses, compared to the 
comparison sample, on every summative assessment in the Pathways' first two academic years. 

We restricted the comparison group for the Statway Mid-Pathway Assessments to a subsample because 
we found a significant relationship between age and performance within this group, where the younger 
half of the sample (ages 18 to 24) performed at a higher level than the older half of the sample (ages 25 
to 30). This may have been due to lag time between the time of their exposure to a statistics course in 
college and that of completing these items. We chose to restrict the ensuing comparison analysis with 
this group to include the younger members of the sample only to help control for this lag and create a 
more valid comparison sample. We used all members of the comparison sample in the other 
comparisons because we found no effect of age. 


© 2014 CARNEGIE FOUNDATION FOR THE ADVANCEMENT OF TEACHING 


Community College Pathways: Summative Assessments 


Table 3 

Compares the Pathways students' and comparison sample's percent of correct responses across items on 
each Pathways summative assessments. 




Pathways Students 

Comparison Sample 

Pathways 

Term 

# of Items ’ 

N % Correct 5 

N 1 % Correct 5 

Difference ' 5 


Statway Mid-Pathway Assessments 


Fall 2011 

44 

829 

61.0 

214+ 

56.2 

+4.9 

Spring 2012 

44 

68 

63.4 

214+ 

56.3 

+7.2 

Fall 2012 

35 

686 

62.3 

214+ 

58.8 

+3.5 

Spring 2013 

35 

358 

62.9 

214+ 

58.8 

+4.1 

Statway End-of- Pathway Assessments 

Spring 2012 

39 

565 

59.9 

500+ 

39.1 

+20.8 

Spring 2013 

32 

671 

62.3 

500+ 

44.9 

+17.4 

Quantway 1 Assessments 

Spring 2012 

37 

372 

65.8 

500+ 

50.0 

+15.8 

Fall 2012 

39 

469 

63.7 

500+ 

50.5 

+13.2 

Spring 2013 

37 

498 

65.2 

500+ 

53.6 

+11.7 


“ Items that were improved during the last faculty review are not included to facilitate exact comparison. 

5 Since multiple forms were used, the standard deviations were not reported; instead, we will use the Rasch person estimates 
for inferential comparisons (see Strother & Yamada, in preparation). 

r Assessment items were pulled from multiple forms administered to the comparison sample, so the N varies slightly by item. 
All forms were completed by at least 500 comparison sample members. The N for Statway Mid-Pathway Assessments is lower 
since only comparison sample members aged 18-24 were used to control for the discovered lag effect. 
d Percentage points. 


Pathways students also consistently performed higher on the items targeting each module within the 
summative assessments. Appendix B displays the main topics of the modules. Appendix C displays the 
performance of the Pathways and comparison sample on the items targeting these modules across all 
summative assessments. 

Relation to Students' Grades 

Students' grades were determined by each individual faculty member and typically included graded 
homework, quiz results, faculty members' judgments, and other faculty-developed tests. As part of 
participation in the Networked Improvement Community, we also asked faculty to include performance 
on the summative assessment as part of their grading system. Each faculty member was free to choose 
the weighting of the summative assessment towards students' final grade. We collected the amount of 
weighting chosen by faculty beginning in fall 2012. Table 4 displays the mean and range of weighting 
faculty used in their courses. We explored the effect of this weighting in the analyses below. 
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Table 4 

Percent of students' grade that was accounted for by the summative assessment, 
averaged across faculty. 


Term 

Faculty 

Mean 

SD Minimum Maximum 

Statway Mid-Pathway Assessments 

Fall 2012 

36 

19.0% 

6.2% 5.0% 

30.0% 

Spring 2013 

17 

18.0% 

7.9% 2.9% 

30.0% 

Statway End-of-Pathway Assessments 

Spring 2013 

39 

15.5% 

6.1% 3.0% 

30.0% 

Quantway 1 Assessments 

Fall 2012“ 

28 

19.9% 

3.9% 5.0% 

25.0% 

Spring 2013 

33 

20.3% 

2.0% 17.0% 

25.0% 


“These summary statistics exclude one outlier that reported 67 percent. However, the 
outlying data were included in the partial correlation and HLM analyses discussed below. 


We found that the grades that Pathway students received in their courses were significantly related to 
the corresponding summative assessment (see Table 5). 


Table 5 

Correlation of Pathways students' summative assessments to their grade (on a 4-point scale) at the end 
of the term. 


Term 

N 

Correlation to 
Students' Grade 

Statway Mid-Pathway Assessments 

Fall 2011 

659 

.61** 

Spring 2012 

25 

.44* 

Fall 2012 

671 

57** 

Spring 2013 

342 

.64** 

Statway End-of-Course Assessments 

Spring 2012 

422 

.60** 

Spring 2013 

653 


Quantway 1 Assessments 

Spring 2012 

252 

.60** 

Fall 2012 

452 

.66** 

Spring 2013 

434 

.66** 


*p < .05 **p<.001 
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These significant relationships persisted after controlling for the percentage of students' grade that was 
accounted for by the summative assessment. We performed partial correlation analysis on all data from 
fall 2012 and spring 2013 that controlled for the weighting of the assessment. All five correlations 
remained significant with nearly identical magnitude; hence we reported the simple bivariate 
correlations here. 

These relationships again persisted when we controlled for the percentage of students' grade that was 
accounted for by the summative assessment in two-level HLM models. We ran HLM analyses with 
students' assessment scores predicting their grade at level 1 and the percent of students' grade 
accounted for by the summative assessment as the level-2 predictor. Students' assessment scores were 
a significant predictor in each analysis, as expected, and the weighting of the assessment did not have a 
significant effect on grade. 

These results show that faculty members' overall assessments of student performance in the course 
were consistent with the summative assessments. This indicates that successful students who received 
higher grades were genuinely achieving higher levels of knowledge, as measured by the independent 
assessment. 
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CONCLUSIONS 

The Pathways have had tremendous success in their first two years (Strother et al, 2013; Van Campen et 
al, 2013). Students' successful completion rates have been consistently near or slightly above 50 
percent, which is approximately triple the success rate of the typical community college program in half 
of the time. 

This report demonstrated that Pathways students are not just completing their path from 
developmental through college-level mathematics faster, but they are doing so while mastering the 
embedded conceptual knowledge at an equal or higher level as compared to other college students who 
have taken college-level mathematics or statistics courses. On virtually every comparison across the 
Pathways summative assessments, the Pathways students consistently performed at or above the level 
of the comparison sample. 

On balance, it is difficult to assess the precise equivalence of the Pathways students and comparison 
sample. We designed test items to align with Pathways content and we administered the assessments at 
the conclusion of the course, which may have favored results from Pathways students. In addition, 
members of the comparison sample took these items some time well after course completion. In terms 
of the first consideration— the content of the items— as shown by the sample items in the appendix, 
these are the kinds of items that one might easily find in any test of college-level statistics and 
mathematics and are validated as such by the numerous expert reviews of program outcomes and 
assessment content built into the development process. So there is little reason to believe that the test 
items especially favored Pathways students. The second consideration, time differences between course 
completion and test administration, is potentially significant, although it is uncertain as to the direction 
of any possible bias. Members of the comparison sample could forget the mathematics they learned 
earlier or have gone on to have subsequent experiences that functioned to consolidate this knowledge. 
However, we controlled for this effect by restricting the comparison sample to ages 18-30 and even 
further to ages 18-24 when there existed any effect of age on performance. 

One could also question whether the incentives for performance were equivalent in both groups. For 
Pathways students the assessment counted as part of their grade; members of the comparison sample 
were offered a very modest stipend for completion. One could argue that the incentives were likely 
greater for Pathways students. Counterbalancing this, the analyses excluded members of the 
comparison sample who did not complete the assessment or whose scores suggested possible random 
guessing. We would expect the latter to advantage the comparison sample. 

The Pathways students' level of knowledge following each term is also reflected by their general success 
in the course. Students' performance on the end-of-term summative assessments was significantly 
correlated to their grades in every term. This demonstrates that the knowledge measured by the 
summative assessment does indeed reflect faculty members' overall evaluation of student performance 
in the course. The results provide further evidence that Pathways students are not just receiving high 
grades and successfully completing the Pathways at a high rate, but that grades in the courses are 
reflective of actual learning. 
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Taking all of this into account and being conservative in drawing inferences, we conclude that these data 
suggest that Pathways students did at least as well as students who had previously taken college-level 
statistics and mathematics at other institutions of higher education. When combined with evidence 
about much higher success rates in the Pathways, as noted above, this provides one more piece of 
evidence supporting a conclusion about the efficacy of the Pathways in advancing mathematical learning 
and success for students who traditionally have found this domain uninteresting, irrelevant, and 
extremely difficult to engage. 
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APPENDIX A 

Sample Summative Assessment Items: Statway 


Learning Outcome: Given the description of a statistical study, determine if it is an observational study 
or an experiment 


A researcher conducted a study to determine if aspirin reduces the chance of a heart attack. He 
randomly assigned 250 patients to take aspirin every day and 250 patients to take a placebo every 
day. After a certain length of time he reported the percentage of heart attacks for the patients 
who took aspirin every day and for those who did not take aspirin every day. 

Is this an observational or an experimental study and why? 

a. The study is observational because it involves random assignment of treatments to 
patients. 

b. The study is observational because it involves observing who had heart attacks. 

c. The study is experimental because it involves random assignment of treatments to 
patients. 

d. The study is experimental because it involves observing who had heart attacks. 


Learning Outcome: Recognize hypothesis testing situations where a chi-square goodness-of-fit test can 
be used to answer a question of interest 


Which of the following scenarios would be an appropriate use of a chi-square goodness-of-fit test? 

a. Researchers want to determine if males spend more on video games than females. 

b. Researchers want to determine if more males eat ice cream than females. 

c. Researchers want to determine if a six-sided die is fair, or has an equal chance of landing 
(16.7%) on each number. 

d. Researchers want to determine if more females prefer to eat their ice cream in a cup than 
males. 
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Learning Outcome: Explain how the relative values of the mean and median are related to the shape 
of the data distribution 


Which histogram MOST LIKELY represents data with a mean that is less than the median? 
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Learning Outcome: Interpret (in context) the slope of the regression line. 


Studies have been done to determine if there is a link between the number of colas (a type of 
carbonated beverage) that American women consume each week and their bone mineral density in 
g/cm 2 . The researcher asked women how many colas they consume each week. The researcher 
also measured the women's bone mineral density. The researcher recorded her data in a 
scatterplot. The following scatterplot shows the relationship between the number of colas 
consumed per week and the bone mineral density along with the least squares regression line. 


y = -0.005x + 0 888 



Number of Colas Consumed Per Week 


Source: Based on data obtained from Katherine Tucker, et al. "Colas, but not other 
carbonated beverages, are associated with low bone mineral density on older women: 
The Framingham Osteoporosis Study." American Journal of Clinical Nutrition 2006 


The slope of the regression line indicates that for every additional cola consumed per week, the 
predicted bone density: 

a) decreases by .005 g/cm 2 . 

b) increases by .005 g/cm 2 . 

c) decreases by .888 g/cm 2 . 

d) increases by .888 g/cm 2 . 
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Learning Outcome: Given a two-way table summarizing a bivariate categorical data set, describe (in 
context) the relationship between the two variables used to construct the table in terms of 
independence or lack of independence. 


Researchers interested in the smoking habits of college students conducted a study. The following 
table shows the data researchers collected: 



Male 

Female 

Total 

Smoker 

260 

78 

338 

Non-smoker 

329 

333 

662 

Total 

589 

411 

1000 


What do the conditional proportions of male smokers and female smokers tell you about the 
smoking habits of college students? 

a. Female college students are less likely to be smokers. 

b. Male college students are less likely to be smokers. 

c. Male and female college students are equally likely to be smokers. 

d. Cannot determine from the information given. 
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Sample Summative Assessment Items: Quantway 


Learning Outcome: Estimate the percentage of a given number and the percentage one number is of 
another number 


The US federal budget in 2011 was approximately $3,360 trillion. Approximately 1.5% of the budget 
was spent on foreign aid. Which of the following is the best estimate for the amount of the US 
federal budget that was spent on foreign aid? 


a. $34 billion 

b. $48 billion 

c. $336 billion 

d. $480 billion 


Learning Outcome: Compute absolute and relative changes 

In January 2011, Sam made a new year's fitness goal. He decided he wanted to be able to work out a 
lot longer on the elliptical trainer. When he started, in January, 2011, he could only spend 20 
minutes on the elliptical trainer before he needed to rest. By July, 2011, he was able to work out for 
35 minutes on the elliptical trainer before he needed to rest. Which of the following expressions 
would give the absolute change in his workout time? 


a. 75% 

b. 43% 

c. 2.5 minutes per month 

d. 7 minutes per year 

e. 15 minutes 
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Learning Outcome: Read and use information presented in a two-way table 


Suppose 3200 women received a mammogram to test for breast cancer. The mammogram provides 
test result information about whether the women do or do not have breast cancer. The table below 
summarizes the mammogram results. The table shows how often the mammogram correctly 
detected the presence of breast cancer in the 3200 women. 



Women with 
Breast Cancer 

Healthy Women (no 
Breast Cancer) 

Total 

Positive test result 

11 

5 

16 

Negative test result 

17 

3167 

3184 

Total 

28 

3172 

3200 


If a woman gets a positive mammogram test result, what is the chance she has breast cancer? 

a. 0.3% 

b. 31% 

c. 40% 

d. 57% 

e. 69% 
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Learning Outcome: Understand and use graphical displays: know that the reference value must be 
known to fully understand a pie graph 


The following two pie graphs display graduation rates for 2012 for two different high schools. At 
Kenwood High, there were 1397 seniors that year, and at Fremont High, there were 643 seniors that 
year. 



Which of the following statements is true about the students who graduated from the two high 
schools? 

a. Kenwood High had more graduates, but a lower percentage of students graduated. 

b. Kenwood High had more graduates, and a higher percentage of students graduated. 

c. Fremont High had more graduates, but a lower percentage of students graduated. 

d. Fremont High had more graduates, and a higher percentage of students graduated. 
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Learning Outcome: Use the compound interest formula for different compounding periods 


Chin has put $5000 into a Certificate of Deposit (CD) that earns a 1.8% Annual Percentage Rate 
(APR) compounded quarterly. The equation for the amount the CD is worth is: 


A = P( 1 + -) 

n 


n*t 


where: 

A = amount the CD is worth 
P = principal (initial amount invested) 
r = Annual Percentage Rate (as a decimal) 
t = time in years 

n = the number of compounding periods in a year 


How much money will Chin have at the end of 5 years? 

a. $5,045 

b. $5,470 

c. $12,059 

d. $25,453 
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APPENDIX B 


Statway Modules (titles) 

Module 1: Types of Statistical Studies and Producing Data 
Module 2: Summarizing Data Graphically and Numerically 
Module 3: Examining Relationships: Quantitative Data 
Module 4: Non-Linear Models 

Module 5: Relationships in Categorical Data with Intro to Probability 

Module 6: Probability and Probability Distributions 

Module 7: Linking Probability to Statistical Inference 

Module 8: Inference for One Proportion 

Module 9: Inference for Two Proportions 

Module 10: Inference for Means 

Module 11: Chi-Square Tests 

Module 12: Mathematical Models 

Quantway 1 Modules (main topics) 

Module 1: Large numbers, estimation and calculation, percentages and ratios 
Module 2: Using ratios, percentages, graphical displays, measures of central tendency 
Module 3: Making conversions, geometric reasoning, formulas and algebraic expressions, creating and 
solving equations 

Module 4: Linear and exponential models 

Quantway 2 Modules (main topics) 

Module 5: Large numbers, estimation and calculation, percentage and ratio, graphical displays (in depth) 
Module 6: Probability and measures of central tendency (in depth) 

Module 7: Linear and exponential models (in depth) 

Module 8: Other types of modeling 
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APPENDIX C 


Table 6 

Compares the Statway students' and comparison sample's percent of correct responses across items on 
each module on the Mid-Pathways summative assessments. 



# of 
Items" 

Fall 2011 

Pathways 

Difference* 6 

# of 
Items" 

Spring 2012 

Pathways 

Difference* 6 

Statway 6 

N* 

Comparison 6 

N 

Statway 6 

N 

Comparison 6 

N 

Module 1 

7 

58.3 

829 

59.8 

214+ 

-1.5 

7 

60.3 

68 

59.8 

214+ 

+0.5 

Module 2 

9 

66.0 

829 

60.7 

214+ 

+5.3 

9 

68.6 

68 

60.7 

214+ 

+8.0 

Module 3 

12 

61.4 

829 

54.3 

214+ 

+7.1 

12 

62.5 

68 

54.3 

214+ 

+8.2 

Module 4 

7 

51.8 

654 

50.0 

214+ 

+1.8 

7 

58.2 

68 

50.0 

214+ 

+8.2 

Module 5 

2 

68.5 

623 

50.5 

214+ 

+18.0 

2 

69.9 

68 

50.5 

214+ 

+19.4 

Module 6 

7 

63.8 

465 

57.7 

214+ 

+6.1 

7 

64.9 

68 

57.7 

214+ 

+7.2 


a Items that were improved during the last faculty review are not included to enable exact comparison. 
f Percentages. 


6 The N varies since faculty were allowed to exclude items that targeted materials they did not cover and students occasionally 
skipped items. 
d Percentage points. 



# of 
Items" 

Fall 2012 

Pathways 

Difference* 6 

# of 
Items" 

Spring 2013 

Pathways 

Difference*' 

Statway 6 

N* 

Comparison 6 

N 

Statway 6 

N* 

Comparison 6 

N 

Module 1 

9 

58.9 

686 

60.2 

214+ 

-1.2 

9 

59.0 

358 

60.2 

214+ 

-1.1 

Module 2 

7 

56.9 

686 

51.5 

214+ 

+5.4 

7 

57.2 

358 

51.5 

214+ 

+5.7 

Module 3 

7 

69.3 

686 

63.9 

214+ 

+5.4 

7 

68.1 

358 

63.9 

214+ 

+4.2 

Module 4 

5 

75.8 

686 

70.3 

214+ 

+5.5 

5 

75.5 

358 

70.3 

214+ 

+5.2 

Module 5 

1 

30.6 

565 

31.5 

214+ 

-0.9 

1 

52.8 

358 

31.5 

214+ 

+21.3 

Module 6 

6 

59.5 

404 

54.2 

214+ 

+5.4 

6 

60.4 

315 

54.2 

214+ 

+6.2 


a Items that were improved during the last faculty review are not included to enable exact comparison. 
b Percentages. 


6 The N varies since faculty were allowed to exclude items that targeted materials they did not cover and students occasionally 
skipped items. 
d Percentage points. 
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Table 7 

Compares the Statway students' and comparison sample's percent of correct responses across items on 
each module on the End-of-Pathways summative assessments. 



# of 
Items 11 

Spring 2012 

Pathways 

Difference^ 

# of 
Items- 

Spring 2013 

Pathways 

Difference^ 

Statway* 

N c 

Comparison* 

N 

Statway* 

N 

Comparison* 

N 

Module 1 

0 

- 

- 

- 

- 

- 

1 

53.4 

671 

48.3 

500+ 

+5.1 

Module 2 

1 

73.6 

454 

60.8 

500+ 

+12.8 

2 

62.7 

671 

56.3 

500+ 

+6.4 

Module 3 

1 

48.2 

444 

44.2 

500+ 

+4.0 

2 

77.0 

671 

65.8 

500+ 

+11.2 

Module 4 

2 

72.4 

463 

53.3 

500+ 

+19.2 

2 

69.4 

671 

65.7 

500+ 

+3.7 

Module 5 

1 

61.4 

425 

47.3 

500+ 

+14.1 

0 

- 

671 

- 

500+ 

- 

Module 6 

3 

58.8 

522 

46.3 

500+ 

+12.5 

3 

66.1 

671 

46.0 

500+ 

+20.1 

Module 7 

1 

52.3 

555 

16.8 

500+ 

+35.5 

2 

61.5 

671 

22.0 

500+ 

+25.5 

Module 8 

9 

57.7 

565 

37.2 

500+ 

+20.5 

8 

69.6 

671 

45.7 

500+ 

+23.9 

Module 9 

1 

57.3 

440 

16.8 

500+ 

+40.5 

2 

68.0 

671 

30.0 

500+ 

+38.0 

Module 10 

6 

58.0 

450 

34.3 

500+ 

+23.7 

6 

44.8 

671 

32.3 

500+ 

+12.6 

Module 11 

3 

55.3 

218 

33.7 

500+ 

+21.5 

1 

39.9 

671 

36.0 

500+ 

+3.9 

Module 12 

5 

65.9 

411 

43.6 

500+ 

+22.3 

3 

69.2 

671 

51.9 

500+ 

+17.3 


a Items that were improved during the last faculty review are not included to enable exact comparison. 
* Percentages. 


c The N varies since faculty were allowed to exclude items that targeted materials they did not cover and students occasionally 
skipped items. 
d Percentage points. 
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Table 8 

Compares the Quantway 1 students' and comparison sample's percent of correct responses across items 
on each module on the summative assessments. 



# of 
Items 0 

Spring 2012 

Pathways 

Difference'* 

# of 
Items' 1 

Fall 2012 

Pathways 

Difference 

Quantway* 

N c 

Comparison* 

N 

Quantway* 

N 

Comparison* 

N 

Module 1 

12 

70.7 

372 

60.4 

500+ 

+ 10.3 

11 

65.5 

469 

61.8 

500+ 

+3.7 

Module 2 

6 

62.3 

372 

46.8 

500+ 

+15.6 

7 

54.8 

469 

47.2 

500+ 

+7.6 

Module 3 

5 

62.8 

358 

39.0 

500+ 

+23.8 

7 

62.6 

469 

43.2 

500+ 

+19.4 

Module 4 

14 

64.1 

372 

46.3 

500+ 

+17.8 

14 

67.3 

469 

46.9 

500+ 

+20.4 


a Items that were improved during the last faculty review are not included to enable exact comparison. 

* Percentages. 

c The N varies since faculty were allowed to exclude items that targeted materials they did not cover and students occasionally 
skipped items. 
d Percentage points. 



# of 
Items 3 

Spring 2013 

Pathways 

Difference 3 

Quantway* 

N 

Comparison* 

N 

Module 1 

10 

71.4 

498 

57.1 

500+ 

+14.4 

Module 2 

6 

64.6 

498 

50.4 

500+ 

+14.2 

Module 3 

7 

55.7 

498 

57.0 

500+ 

-1.3 

Module 4 

14 

65.8 

498 

50.7 

500+ 

+15.1 


a Items that were improved during the last faculty review are not included to enable exact comparison. 
b Percentages. 
c Percentage points. 
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